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Abstract 


oventry University Institution Wide Language Programme 
C.. offers beginners language learning modules to 
approximately 3,000 students. Each module is taught over 11 weeks 
for a total of one hour and 40 minutes and two of the weeks are used 
for in class tests, so tutors generally agree that students need to practise 
their skills outside the classroom in order to pass the module. One way 
of doing this was to use the language learning app, Duolingo, which 
helps students to gradually broaden their linguistic, professional 
competence, and digital fluency through increased learner autonomy. 
How effective this approach is, however, has never been measured. 
The project ran from January to April 2018 and explored whether a 
correlation exists between regular use of the app by IWLP learners 
at Common European Framework of Reference (CEFR) level Al of 
French, German, Italian, Portuguese, and Spanish, and achieving a 
high formal coursework assessment mark. The views of both learners 
and tutors of their experiences of using the software, and the tutor 
tools provided by Duolingo Schools, will also be canvassed. A virtual 
classroom was set up within Duolingo Schools for each participating 
cohort of students and they were encouraged to use the app on a 
regular basis. The tutor tools allowed the monitoring of how many 
days learners were active, how many lessons they completed, how 
many courses they completed, and how many points they were 


awarded while using Duolingo. These statistics were compared with 
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the overall formal assessment grades on modules and further data 
was collected at the end of the project from a representative sample 
of learners (182) and tutors (ten) to establish whether the software 


influenced module marks. 
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1. Introduction 


Language learning is increasingly taking place in a variety of formal and 
informal settings where mobile technologies and gamification have become more 
integrated into the language learning process. The vast number of opportunities 
to practise language learning outside the classroom provides instructors with the 
challenge of directing students towards resources most effective for their needs. 
One of these resources is Duolingo, a free language platform available both on 
PC and mobile. The site purports to teach nearly 2,000 vocabulary items for 
each language it offers and that studying via the app students are able to reach a 
vocabulary level of B1 on the CEFR if they complete all the activities available. 


Duolingo delivers its content via a series of translation exercises, multiple choice 
quizzes, flashcards, word-pairing, and translating unknown words through 
clicking on them. In 2015, Duolingo Schools was launched which allows tutors 
to create virtual classrooms where they can assign specific homework tasks and 
challenges to students and monitor their progress. Alternatively, students can be 
challenged to study whatever they want on the platform in order to accumulate 
a target number of Duolingo Experience (XP) points which are awarded for 
successfully completing tasks on the site. 


The self-directed learning skills students need to succeed on Duolingo help to 
develop professional competences essential for success on the job market such 
as autonomy, responsibility, motivation, self-efficacy, management, reflection, 
and persistence. 
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To contextualise this study, it is essential to understand that all UK universities 
are measured on the first jobs that their students acquire after graduation and 
therefore the employability skills of the graduates have become one of the most 
fundamental priorities in higher education (Routes into Languages, 2019). 


In response to this, Coventry University implemented several new strategies 
to emphasise and stress the importance of education for employability. One 
of these strategies involved incorporating the teaching of employability skills 
into WLP (Cervi-Wilson & Brick, 2016). IWLPs typically cater for students 
who want to study a language alongside their main degree subject. Uniquely, 
Coventry University offers language modules to undergraduate students as part 
of a scheme called Add+Vantage. 


Typically, there are 24 students in each class and each class lasts for one hour 
and 40 minutes in total. The classes run over 11 weeks during which two weeks 
are used for assessment, leaving only nine full weeks of teaching. Much of the 
first class is taken up explaining how the module and assessments are organised 
and explaining to learners the procedures they are obliged to adhere to. This 
reduces the amount of contact time even further. As a result of this, students 
are encouraged to deepen their learning by practising what they have learnt in 
the classroom using a variety of self-directed learning tools such as Duolingo. 
Analysing students’ use of language learning apps is extremely beneficial for 
instructors to understand where and how students focus their efforts outside 
the classroom. This study aims to establish whether or not widespread use of 
Duolingo has an impact on students’ in class test performance in their language 
learning modules. 


There have been numerous studies about Duolingo (Botero, Questier, & Zhu, 
2018; Garcia, 2013; Hermoso-Gomez, 2016; Krashen, 2014; Rosell-Aguilar, 
2017; Teske, 2017; to name but a few) all of which have been critical of the 
product in numerous ways due to its repetitive nature, behaviouristic approach, 
the audiolingual method it adheres to in part, and the lack of opportunity for 
speaking practice. Its use of unauthentic voices for listening practice has also 
been widely criticised (Teske, 2017). 
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Recent studies in Mobile Assisted Language Learning (MALL) literature 
(Chwo, Marek, & Wu, 2018) have revealed three common weaknesses. One is 
the discrepancy between how instructors think students use their devices and 
how they actually use them. The second relates to issues including motivation. 
This generally relates to the perceptions of students 


“concerning the usefulness of the MALL technology and lessons, and 
their resulting level of engagement in the MALL assignments, lessons, 
or activities. The third surrounds the short duration of MALL studies 
and poor research design” (Chwo et al., 2018, p. 62). 


This study allowed students complete freedom to use the platform for self- 
directed learning and students were motivated, to some extent, by the link to 
achieving success on their IWLP module. What the study could not address 
was the duration of time students engaged in with the platform due to the fact 
that their IWLP course only lasted 11 weeks. In terms of research design, both 
quantitative and qualitative data were collected in order to triangulate results. 


Ze Methodology 


The participants were IWLP students at Coventry University registered on degree 
courses across all disciplines and they are all expected to spend approximately 
ten hours per week on self-directed study. Students were introduced to the 
different features Duolingo offers and were asked to make an informed decision 
whether they wanted to be part of the project. Students were expected to monitor 
their own activity and interference by staff was kept to a minimum. There was 
no control group involved in this study due to ethical issues. 


All staff teaching on European language Add+Vantage modules were invited 
to a meeting where the project was outlined to them and the process of setting 
up a classroom in Duolingo Schools, a separate platform which allows tutors to 
set homework and monitor student progress, was explained. Ethics forms were 
also distributed, but out of 35 members of staff teaching on the Add+Vantage 
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language provision only 11 tutors were recruited, representing 182 students 
studying French, German, Spanish, and Italian at CEFR Al level (see Figure | 
for breakdown). 


Figure 1. Project participant language breakdown 


= French 
= German 
italian 


= Spanish 


Some staff were already using Duolingo Schools, while others were interested 
in implementing it in their teaching. Anecdotal evidence suggested that students 
who regularly used Duolingo to practise what they had learnt in class achieved 
higher scores than those who had not engaged with the app. Therefore, the 
project sought to establish whether there was in fact a correlation between 
extensive use of Duolingo and achieving higher grades on the two courseworks 
that all Add+Vantage students are required to take. The data was collected for 
a period of 11 weeks which is the same period of time they spent studying on 
the Add+vantage programme. The tutor tools within Duolingo provided the 
necessary data to map against student performance. The data was generated in 
the form of XP points which students earned when they successfully completed 
tasks on the platform (Munday, 2016; Teske, 2017). In addition to this, each 
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individual tutor set up small focus groups with their students to gather data 
regarding their experience of being involved in the project. 


3. Results 


At the end of the project, two sets of data were generated. The first one 
represented the marks that students had achieved by completing the required 
formal assessment for their module. The second set of data was generated by 
the Duolingo Schools platform and provided the exact number of XP points 
each participant had gathered during the project. The data was then merged 
to provide a graph correlating XP points against the overall coursework mark 
achieved by each participant. Details of these are provided in Figure 2, Figure 3, 
and Figure 4. The data was divided into three sections based on the number of 
XP points collected. Figure 2 shows the number of students who achieved up 
to 500 XP points, Figure 3 shows 500-1,000 and Figure 4 shows 1,000+. The 
quantitative results reveal some correlation between the number of XP points 
accumulated on Duolingo and the overall coursework mark achieved on the 
module. 


Figure 2 shows that the vast majority of participants achieved fewer than 500 XP 
points throughout the duration of the project. The graph does not reveal a strong 
correlation between using Duolingo and performing well on the courseworks. In 
fact, 15% of participants achieved a mark above 80% and 4% failed the module. 


Figure 3 shows that 13% of participants achieved between 500 and 1,000 XP 
points and of those 33% achieved a score above 80% on their coursework and 
all participants passed the module. However, 61% of this group scored between 
40% and 60% on their coursework and only one failure was recorded. 


The sample in Figure 4 is numerically far smaller than the other two groups but 
shows that 43% of participants who accumulated over 1,000 XP points achieved 
a module mark of over 80%. However, 21% of this group achieved a module 
mark ranging from 40% to 60% and one failure was recorded. 
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Figure 2. Participants scoring less than 500 XP points 
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Figure 3. Participants scoring between 500 and 999 XP points 
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Figure 4. Participants scoring more than 999 XP points 
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Overall the statistics show a slight correlation between the accumulation of XP 
points on Duolingo and achieving over 80% in the module courseworks. 


The illustrative qualitative data revealed some further trends. Three of the 
participants were negative about Duolingo, describing it as boring and repetitive 
and criticising the lack of authentic audio available in the app. However, most 
of the responses collected from the focus group sessions revealed a positive 
experience using Duolingo, praising numerous features including the email 
reminders, the regular testing, the variety of activities and, in particular, how 
effective they found it when used in conjunction with formal learning undertaken 
in a classroom setting. 


The following statements are representative of the comments collected from 
students by the tutors and form the qualitative results of the research: 


“Not many speaking activities and the voices aren’t authentic” 
(Student A). 
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“It’s boring at times and a bit repetitive” (Student B). 


“T had tried to use it to learn another language without any assistance, 
but I did not find it quite as useful then” (Student C). 


“Duolingo sends regular reminders” (Student D). 


“It’s a good opportunity to revise before a test’ (Student E). 


“There is so much variety. It gives us additional skills and experience” 
(Student F). 


“In my experience Duolingo was quite useful, but more so when it was 
used in conjunction with a class or with some prior knowledge of the 
language” (Student G). 


“When I used it for Italian, in combination with what I’d learned in 
class, I thought it was quite effective because it used active recall 
revision tactics, which I know help me learn best” (Student H). 


“The fact that the things I had got wrong were tested again at the end of 
each ‘session’” (Student I). 


“It was also very helpful to see where my strengths and weaknesses 
were with learning Italian” (Student J). 


“Overall, I thought Duolingo was a useful tool for me in studying 
Italian. I don’t know if it helped me to improve my grade because I 
used it in combination with other tools, such as languagesonline, my 
notes from the class and the textbook. Some ‘sessions’ in Duolingo 
were obviously more relevant to the class content than others, so I 
don’t know if there was a correlation between my Duolingo use and 
my marks” (Student K). 
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4. Conclusions 


The quantitative data revealed a slight correlation between obtaining over 
100 Duolingo XP points over an 11-week period and achieving an above average 
mark in the module assessment. The qualitative data revealed differing attitudes 
amongst participants, but the majority were overall positive about the app when 
they used the tool in conjunction with regular attendance at a language learning 
class. 


The study itself has its limitations partly due to the relatively small sample and 
the duration, common to many MALL studies as mentioned above (Chwo et al., 
2018). Even though 456 students were approached to take part in the project, 
only 182 (40%) agreed to share their data with us. It was also impossible to 
isolate Duolingo as the only source of extra-curricular study. The data collected 
was only based on the accumulation of XP points, which only provided an 
overview of the amount of activities participants undertook, rather than a more 
focussed approach on specific tasks assigned by the tutor within the app itself. It 
was also apparent that some tutors encouraged the students to use the app on a 
regular basis more than others. 


Finally, we can conclude that Duolingo may be used most effectively as a 
supplementary language learning tool to consolidate and deepen knowledge 
acquired in a classroom setting rather than as a sole source. 
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